{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Los Alamos National Lab Data Preparation\n",
    "*Source:* [LANL dataset](https://csr.lanl.gov/data/cyber1/)  \n",
    "\n",
    "This data set represents 58 consecutive days of de-identified event data collected from five sources within Los Alamos National Laboratory’s corporate, internal computer network.\n",
    "\n",
    "Only the auth.txt file is used in our current work, as all red team activity appearing in the data correspond exclusively to authentication events. Future work includes utilizing additional data streams (namely; proc.txt.gz). We perform a pre-processing step on the file redteam.txt.gz so that its log lines are expanded to match the full log line which they correspond to in the auth.txt.gz file. This adds general convenience, and speeds up the process of querying to find out if a given log line is malicious.\n",
    "\n",
    "This notebook outlines methods used for translating log lines into integer vectors which can be acted on by event level models. Note that the scripts in **/safekit/features/lanl** can also be used standalone to accomplish the same translation, but only operate on the auth.txt data file.\n",
    "\n",
    "### Character Level\n",
    "----\n",
    "*note: /safekit/features/lanl/char_feats.py*  \n",
    "At the character level, the ascii value for each character in a log line is used as a token in the input sequence for the model.\n",
    "\n",
    "The translation used is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate_line(string, pad_len):\n",
    "    return \"0 \" + \" \".join([str(ord(c) - 30) for c in string]) + \" 1 \" + \" \".join([\"0\"] * pad_len) + \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where **string** is a log line to be translated, and **pad_len** is the number of 0's to append so that the length of the translated string has the same number of characters as the longest log line in the dataset (character-wise). **0** and **1** are used to describe start and end of the translated sentence. \n",
    "\n",
    "The length of the max log line can be obtained using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "129\n"
     ]
    }
   ],
   "source": [
    "max_len = 0\n",
    "with open(\"auth_proc.txt\", \"r\") as infile:\n",
    "    for line in infile:\n",
    "        tmp = line.strip().split(',')\n",
    "        line_minus_time = tmp[0] + ',' + ','.join(tmp[2:])\n",
    "        if len(line_minus_time) > max_len:\n",
    "            max_len = len(line)\n",
    "print (max_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We chose to filter out weekends for this dataset, as they capture little activity. These days are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weekend_days = [3, 4, 10, 11, 17, 18, 24, 25, 31, 32, 38, 39, 45, 46, 47, 52, 53]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also convenient to keep track of which lines are in fact red events. Note that labels are not used during unsupervised training, this step is included to simplify the evaluation process later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"redevents.txt\", 'r') as red:\n",
    "    redevents = set(red.readlines())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**redevents.txt** contains all of the red team log lines verbatim from auth.txt.   \n",
    "It is now possible to parse the data file, reading in (raw) and writing out (translated) log lines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "with open(\"auth_proc.txt\", 'r') as infile, open(\"ap_char_feats.txt\", 'w') as outfile:\n",
    "    outfile.write('line_number second day user red seq_len start_sentence\\n') # header\n",
    "    infile.readline()\n",
    "    for line_num, line in enumerate(infile):\n",
    "        tmp = line.strip().split(',')\n",
    "        line_minus_time = tmpline_minus_time = tmp[0] + ',' + ','.join(tmp[2:])[0] + ',' + ','.join(tmp[2:])\n",
    "        diff = max_len - len(line_minus_time)\n",
    "        raw_line = line.split(\",\")\n",
    "        sec = raw_line[1]\n",
    "        user = raw_line[2].strip().split('@')[0]\n",
    "        day = math.floor(int(sec)/86400)\n",
    "        red = 0\n",
    "        line_minus_event = \",\".join(raw_line[1:])\n",
    "        red += int(line_minus_event in redevents) # 1 if line is red event\n",
    "        if user.startswith('U') and day not in weekend_days:\n",
    "            translated_line = translate_line(line_minus_time, 120) # diff\n",
    "            outfile.write(\"%s %s %s %s %s %s %s\" % (line_num, sec, day, \n",
    "                                                    user.replace(\"U\", \"\"), \n",
    "                                                    red, len(line_minus_time) + 1, translated_line))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final preprocessing step is to split the translated data into multiple files; one for each day. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.mkdir('./char_feats')\n",
    "with open('./ap_char_feats.txt', 'r') as data:\n",
    "    current_day = '0'\n",
    "    outfile = open('./char_feats/' + current_day + '.txt', 'w')\n",
    "    for line in data:\n",
    "        larray = line.strip().split(' ')\n",
    "        if int(larray[2]) == int(current_day):\n",
    "            outfile.write(line)\n",
    "        else:\n",
    "            outfile.close()\n",
    "            current_day = larray[2]\n",
    "            outfile = open('./char_feats/' + current_day + '.txt', 'w')\n",
    "            outfile.write(line)\n",
    "    outfile.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The char_feats folder can now be passed to the tiered or simple language model.  \n",
    "The config (data spec) or this experiment is shown below (write as string to json file)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">{**\"sentence_length\"**: 129,  \n",
    "    **\"token_set_size\"**: 96,  \n",
    "    **\"num_days\"**: 30,  \n",
    "    **\"test_files\"**: [\"0head.txt\", \"1head.txt\", \"2head.txt\"],  \n",
    "    **\"weekend_days\"**: [3, 4, 10, 11, 17, 18, 24, 25]}  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Level\n",
    "----\n",
    "Instead of using the ascii values of individual characters, this approach operates on the features of a log line as if they were a sequence. In other words, each log line is split on \",\" and each index in the resulting array corresponds to a timestep in the input sequence of the event level model.\n",
    "\n",
    "To map the token strings to integer values, a vocabulary is constructed for the dataset. Any tokens encountered during evaluation which were not present in the initial data are mapped to a common \"out of vocabulary\" (OOV) value during translation. If the number of unique tokens within the data is known to be prohibitively large, a count dictionary can be used to infer an arbitrary cutoff which maps the least likely tokens in the data to the OOV integer. Eg; all tokens which appear less than 5 times in the data map to the OOV token during translation.\n",
    "\n",
    "Note that in our AICS paper we have seperate OOV tokens for user, pc, and domain tokens.  \n",
    "In the following code, every unique token is included in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = 4 # <sos> : 1, <eos>: 2, auth_event: 3, proc_event: 4\n",
    "vocab = {\"OOV\": \"0\", \"<sos>\": \"1\", \"<eos>\": \"2\",\"auth_event\": \"3\", \"proc_event\": \"4\"}\n",
    "\n",
    "def lookup(key):\n",
    "    global index, vocab\n",
    "    if key in vocab:\n",
    "        return vocab[key]\n",
    "    else:\n",
    "        index += 1\n",
    "        vocab[key] = str(index)\n",
    "        return str(index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Translated log lines should be padded out to the maximum length fields_list. In the case of LANL, the proc events are longer and contain 11 tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate(fields_list):\n",
    "    translated_list = list(map(lookup, fields_list))\n",
    "    while len(translated_list) < 11:\n",
    "        translated_list.append(\"0\")\n",
    "    translated_list.insert(0, \"1\")  # <sos>\n",
    "    translated_list.append(\"2\")  # <eos>\n",
    "    translated_list.insert(0, str(len(translated_list)))  # sentence len\n",
    "    return translated_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We chose to filter out weekends for this dataset, as they capture little activity. It is also convenient to keep track of which lines are in fact red events. Note that labels are not used during unsupervised training, this step is included to simplify the evaluation process later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weekend_days = [3, 4, 10, 11, 17, 18, 24, 25, 31, 32, 38, 39, 45, 46, 47, 52, 53]\n",
    "\n",
    "with open(\"redevents.txt\", 'r') as red:\n",
    "    redevents = set(red.readlines())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since OOV cutoffs are not being considered, the translation can be done in a single pass over the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "with open(\"auth_proc.txt\", 'r') as infile, open(\"ap_word_feats.txt\", 'w') as outfile:\n",
    "    outfile.write('line_number second day user red sentence_len translated_line padding \\n')\n",
    "    for line_num, line in enumerate(infile):\n",
    "        line = line.strip()\n",
    "        fields = line.replace(\"@\", \",\").replace(\"$\", \"\").split(\",\")\n",
    "        sec = fields[1]\n",
    "        translated = translate(fields[0:1] + fields[2:])\n",
    "        user = fields[2]\n",
    "        day = math.floor(int(sec)/86400)\n",
    "        red = 0\n",
    "        red += int(line in redevents)\n",
    "        if user.startswith('U') and day not in weekend_days:\n",
    "            outfile.write(\"%s %s %s %s %s %s\" % (line_num, sec, day, user.replace(\"U\", \"\"), \n",
    "                                                 red, \" \".join(translated)))\n",
    "print(len(vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final preprocessing step is to split the translated data into multiple files; one for each day. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.mkdir('./word_feats')\n",
    "with open('./ap_word_feats.txt', 'r') as data:\n",
    "    current_day = '0'\n",
    "    outfile = open('./word_feats/' + current_day + '.txt', 'w')\n",
    "    for line in data:\n",
    "        larray = line.strip().split(' ')\n",
    "        if int(larray[2]) == int(current_day):\n",
    "            outfile.write(line)\n",
    "        else:\n",
    "            outfile.close()\n",
    "            current_day = larray[2]\n",
    "            outfile = open('./word_feats/' + current_day + '.txt', 'w')\n",
    "            outfile.write(line)\n",
    "    outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word_feats directory can now be passed to the tiered or simple language model.   \n",
    "The config json (data spec) for the expiriment is shown below (write as string to json file)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">{**\"sentence_length\"**: 12,   \n",
    "    **\"token_set_size\"**: len(vocab),  \n",
    "    **\"num_days\"**: 30,  \n",
    "    **\"test_files\"**: [\"0head.txt\", \"1head.txt\", \"2head.txt\"],  \n",
    "    **\"weekend_days\"**: [3, 4, 10, 11, 17, 18, 24, 25]}  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
